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METHOD AND APPARATUS 
FOR AUTOMATIC APPLICATION AND 
MONITORING OF A STRUCTURE TO BE 
APPLIED ONTO A SUBSTRATE 



fOOO^jOOOll The present application relates to and claims priority from 
European Application Serial No. PCT/EP2004/007964 filed July 16, 2004, titled 
"METHOD FOR AUTOMATIC APPLICATION AND MONITORING OF A 
STRUCTURE TO BE APPLIED ONTO A SUBSTRATE, AND DEVICE 
THEREFORE", the complete subject matter of which is hereby expressly incorporated 
in its entirety. 

[0002] The present invention relates to a method for automatic application and 
monitoring of a structure to be applied onto a substrate, and a corresponding apparatus 
th e r e fore . 

[0003] For automatic application and monitoring of a structure to be applied onto 
a substrate, it has been customary to carry out optical measurements , wh e r e by . For 
automatic application and monitoring; of a structure, frequently various systems for fully 
automatic testing of the structure, including adhesive and sealing agent lines T are used-for 
this purpos e .^ For this purpose, multiple video cameras are directed at the structure to be 
recognized, whereby, in addition, an illumination module serving to generate a contrast- 
rich camera image is required. 

[0004] In order to be able to monitor an adhesive line and/or adhesive trail while 
it is being applied, it is necessary to teach-in a reference adhesive trail-JLeT^ to have the 
camera or cameras scan the reference adhesive trails) in order to calculate therefrom 
corresponding parameters on which the assessment of the applied adhesive trails is based 
subsequently based . 

[0005] However, individual components are not always supplied to the same 
position of the application facility and/or the apparatus for automatic application and 
monitoring by means of the suppK^ supplier of the technology - Moreover, application of 
an adhesive trail to a fold or joining seam requires correction of the tolerances of the 
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individual components and/or correction of the position of the individual joining seams 
and/or folds. 

[0006] Moreover, there is a need for a method for automatic application and 
monitoring of a structure to be applied onto a substrate, preferably an adhesive agent trail 
and/or an adhesive trai l whereby . In addition, the application structure and/or adhesive 
trail is monitored at ahigh accuracy while it is being applied. 

[0007] It is therefore the object of the present invention to provide a method for 
automatic application and monitoring of a structure to be applied onto a substrate 7 
wh e r e by th e . The application structure and/or adhesive trail is monitored at a_high 
accuracy while it is being applied , and wh e r e by automatic . Automatic control of the 
application facility-and/^or positional correction- with regard to positional tolerances of 
the individual components and/or tolerances of the joining seams or similar structures— is 
facilitated. 

[0008] Moreover, it is an object of the present invention to provide a suitable 
apparatus for carrying out the method according to the invention. 

[0010] Th e s e obj e cts ar e m e t with r e gard to th e m e thod by the features of claims 
1 and with r e gard to th e apparatus by th e f e atur e s of claim 15. 

[0009] According to the invention, a method for automatic application and 
monitoring of an adhesive trail onto a substrate aftd/or component, in particular a fold or 
joining seam, is proposed , wh e reby . A reference edge or a reference edge and/or 
r e f e r e nc e seam is determined by a first camera^ in ajeading direction of the application 
facility, in order to control aad/or regulate the application facility according to the 
reference edge by th e r e cord e d imag e s of th e first cam e ra. ^ Simultaneousl y and/or 
directly after applying the adhesive trail onto the substrate mk>r the fold or location 
where components abut, a second camera carries out online monitoring of the applied 
adhesive trail in availing direction-_[i.e. the adhesive trail is applied onto the substrate 
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and then the second camera checks the quality of the adhesive trail that was just appliedy}. 
According to the invention, thts- control or regulation of the application facility according 
to the reference edge facilitates concurrent seam application guidance for two 
components to be glued together and online monitoring of adhesive application and/or 
sealing agent application.^ Thus is achi e v e d ^ a reduction of the sealing agent applied^ 
srnee- is achieved, because the seam application guidance and simultaneous control 
necessitate the use of less material due to the compensation of tolerances. 

[0010] Further advantag e ou s d e v e lopm e nt s ar e evid e nt from th e subclaims. 
Accordingly, for a three-dimensional positional correction with regard to positional 
tolerances of the individual components and/or tolerances of joining seams, it is 
advantageous for the reference contour and/or a feature to be determined by at least two 
cameras , in ord e r to . The two cameras carry out a three-dimensional positional 
correction for the application facility by means of the stereometry procedure. 

[001 1] It is also advantageous if the two cameras record the substrate, a section of 
the component or one or more components in the form of a full image or ajarge image, 
whereby the full images or large images of the two cameras comprise an overlapping area 
in ajeading direction , and wh e r e by th e . The three-dimensional recognition of reference 
contour position r e sultin g results in the overlapping area is- to be used for gross adjustment 
of the application facility prior to applying the structure.- In this context, corresponding 
correction values are transmitted to the application facility and/or the- to a robot in order 
to shift it s coordinat e the robot's coordinant system for the application of the adhesive 
agent. 

[0012] If a projection is made onto the area of the reference contour for thr ee- 
dim e n s ional analysi s , in particular (e.g., if one or more laser lines are applied onto the 
substrate in the form of a projection^} then a three-dimensional analysis of the profile 
with regard to the height and contour of arbitrary components can be facilitated-evea.. 
Even though this the reference contour is not analyzable by common image processing 
without an additional projection. 
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[0013] Moreover, it is particularly advantag e ous a particular advantage is if the 
reference contour is determined just-by a first camera in ajeading direction to regulate 
the progression of the structure to be applied according to the reference contourr-and 
wher e by th e . The first camera records just-a strip of the image for online regulation of 
the application of the adhesive structure - By means of this partial scan an4/or partial 
read-out of the image recording chip, only small data streams need to be processed such 
that the image recording rate can be increased several-fold - In this context, the images 
are recorded at defined fixed time intervals and are independent of the speed of the 
application facility aadA>r the robot speed. 

[0014] If the second camera uses just a strip of the image for online monitoring of 
the applied structure, the adhesive application can proceed at high a higher speed and th e. 
The seam application guidance can proceed at high a higher speed in an online fashion? 
smee because both cameras facilitate high-frequency image recording and rapid analysis 
with just one sensor with two cameras.- In this context, a reference edge is determined in 
ajeading direction parallel to the online inspection of the track applied sealing agent 
track appli e d, and the diff e r e nc e . Difference values are transmitted to the robot for 
correction of the track such that the accuracy of the application of the sealing agent 
application can be incr e as e d significantly incr e as e d and a reduction of the material 
needs needed is attained.- Due to this only partial read-out of the image recording chip of 
the individual cameras, images of all cameras can be captured synchronously, in parallel, 
and simultaneously. 

[0015] According to an advantageous embodiment, the strips of the images of the 
cameras are recorded to form a single sequence of images and wh e r e by the . The image 
recording rate is increased in line with the data reduction achieved by recording jus^a 
strip of the image in order to increase the speed of the automatic application and 
monitoring of the sealing agent application - By storing a single sequence of images for 
all the cameras, the respective images of the individual cameras can be assigned 
according to the travel of the application facility as a function of location. 

4 



Atty. Dkt. No.: QU01H10/P-W01/US (22589-13) 

Express Mail No.: EB 270750837 US 

SUBSTITUTE SPECIFICATION MARKED UP 

[0016] If each camera uses only a part, in particular approx imately a third, 
fourth or fifth, of the picture lines as a^strip of the image, the image recording rate is 
multiplied accordingly, m- In particular , the image recording rate is multiplied essentially 
three-fold, four-fold or five-fold. 

[0017] Moreover, it is advantageous for a parameterization and a recording of the 
application track to proceed in a single image recording run, whereby the images of all 
cameras are stored in a sequence of images. 

[0018] According to the invention, the stored sequence of images uses- use the 
robot travel path and/or ., the robot travel time-er a the robot coordinat e s x oordinants as 
well as the position, the contrast, the gray scale value-et^ color value, the widths and the 
quality of the applied structure for parameterization. 

[0019] Sinee Because only a small amount of data has to be included in the 
parameterization calculation and because of the high image recording rate, it is f e asibl e to 
reeefd -recording comparably short partial sections of the sealing agent application and of 
the reference contour and/or joining seam , which ar e , for e xampl e (e.g. , between 1 mm 
and 3 mm 7 in length s is feasible. Moreover, it is advantageous to store the structure to be 
applied by parameterization essentially in the form of a vector chain , wh e reby a . A high 
image recording rate and short partial sections of essentially between 0.5mm and 4mm, in 
particular between 1 and 3mm, are used - The vectorization is advantageous in that the 
adhesive trail in the form of a vector chain can be stored in a camera-transcending global 
coordinate system.^ In contrast, traditionally, only a local camera image-oriented 
coordinate system is being used.- The invention thus facilitates that a switch of the sensor 
head only necessitates recalibration and/or new calibration without having to teach-in the 
adhesive trail again. 

[0020] According to another advantageous embodiment, it an advantage is 
advantageous to use three cameras, whereby each camera is used at*d/or can be used both 
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for regulation in leading direction according to the reference contour and for monitoring 
of the applied structure in ajrailing direction , wh e r e by th e . The three cameras each 
comprise an overlapping area to the adjacent camera on a circular line._ As a result, a 
sensor with three cameras can be attached to be fixed on the application facility, since 
each individual camera can assume both the regulation of seam application guidance and 
online monitoring of sealing agent application - Advantageously, the angle values of the 
circular line from 0° to 360° form a global coordinate system , wher e by a . A segment of 
the circular line is assigned to the images of the individual cameras in order to carry out 
on thisthe circular line either the seam application guidance or the monitoring of the 
sealing agent.- As a result, two of the three cameras are always active for analysis, i.e. 
one for seam application guidance and one other for the monitoring of sealing agent 
application. 

[0021] Another advantage is that an automatic switch is made, when the reference 
contour or the adhesive trail progresses from one camera to the next camer a, i.e. in that . 
For example, the activation is transferred from the one camera to the other camera when 
the application structure or the reference contour progresses from the segment of the 
circular line of the one camera via the overlapping area to the segment of the circular line 
of another camera. 

[0022] Since the images are recorded very shortly one after the other (e .g., e very 
0.5 to 4 ? mm; in particular^ 1 to 3 mm), it can be assumed that the position of the adhesive 
trail andA>r joining edge cannot change too strongly^ which significantly improves the 
significance and/or reliability of the a- jpriori knowledge such that it- the location of the 
track can be predicted where th e track will be locat e d. ,. One positive effect of thk -such a 
priori knowledge is that the computer can recognize fully automatically the position of 
the track even in the absence of the expert knowledge of a human being, since the 
computer already knows approximately where the track will be progressing in the next 
image - This allows the search area to be reduced and the speed of analysis to be 
increased. 
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[0023] According to the present invention i s provid e d an apparatus is provided 
for automatic application and monitoring of a structure to be applied onto a substrate, 
preferably an adhesive line or adhesive trail, for carrying out the method according to the 
invention , whereby at . At least one illumination module and one sensor unit are 
provided , and wh e r e by th e . The sensor unit is made up of at least two cameras that are 
provided around an application facility for applying the structure to be applied onto the 
substrate and are arranged on this the application facility such that at least one camera is 
provided in a_leading direction for regulation of the application facility by means of a 
reference contour and at least one camera is provided in a trailing direction for 
simultaneous online monitoring of the structure applied onto the substrate.— The 
apparatus^ according to the invention^ can therefore be used, for example, to guide a seam 
as preference contour for the control of the application facility aftd/or robot contro l and . 
The apparatus can simultaneously carry out an online control of sealing agent application, 
such that less material is used in sealing agent application since the width of the adhesive 
agent track can be reduced due to the guidance of the application facility. 

[0024] If the optical axes of the individual cameras essentially intersect, in the 
direction of view? (e.g. the axial longitudinal axis of the application facility): or if the 
optical axes of the individual cameras are directed to be parallel to each otherr-an d (e.g., 
in particular are directed to be perpendicular to the substrate , it is advantag e ous ) an 
advantage according to a development of this type is that a narrow area around the 
application facility can be monitored at ^suitable resolution and ahigh image recording 
rate. 

[0025] According to a preferred embodiment, the individual cameras, in 
particular 3 cameras, are arranged at equal distances from each other in the direction of 
the circumference. 

[0026] Advantageously, the individual cameras interact with each other such that 
the images of the cameras are stored in a sequence of images, whereby these images are 
composed by the software from the three partial sections of the individual cameras that 
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were recorded synchronously and captured in parallel. 

[0027] If a projection facility projecting one or more features, in particular strips, 
onto the substrate for the three-dimensional analysis is provided on the application 
facility, arbitrary components can be used for correction and/or adjustment of the 
application facility prior to applying the structure. 

[0028] According to a preferred embodiment, the projection facility emits one or 
more laser lines for a three-dimensional profile analysis.- Arranging at least two 
projection facilities around the application facility facilitates a_gap-free three-dimensional 
analysis around the application facility , wh e r e by th e . The analysis of sealing agent 
height and sealing agent contour^ as well as position and widths can be carried out 
according to the principle of triangulation by means of image processing. 

[0029] According to a development of an invention, the cameras are arranged 
around the application facility . The cameras are arranged such that at least an essentially 
circular edge scan (e.g. , in particular in the form of a circular caliper^) is formed whose 
center is formed by the application facility of the structure.- In this context, one or more 
circular calipers can be used that facilitate the determination of the edge of the adhesive 
trail on a circular line. 

[0030] According to a preferred embodiment, the individual cameras comprise an 
overlapping area of 30° to 900° each, in particular essentially 60°, relative to the next 
camera. This overlapping area facilitates fully automatic switching between neighboring 
cameras when the adhesive trail progresses from the monitoring area of one camera to the 
monitoring area of the next , sinc e the camera. The selection of the camera is not bound 
to the robot position or to a time component, but rather always refers to the actual 
inspection results , i.e. is . For instance, the inspection results are based on the 
arrangement on the circular line of the circular caliper and/or the global coordinate 
system formed thereby. 
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[0031] Moreover, it is advantageous for the illumination module to be made up of 
light emitting diodes ( LEDs), in particular infrared LEDs, ultra-violet LEDs ( UV LEDs) 
or red-green-blue LEDs ( RGB LEDs). 

{G034}-Moreover, it is of one advantage isto use a calibrating disc with individual 
form elements for calibrating the individual cameras for the assignment of the angle 
assignmen t wher e by . The said form elements comprise, in particular, an angle distance 
of essentially 10°.- This allows for assignment of the scaling factor, angle assignment, 
and center^ as well as radius of the search circle for the individual cameras.- According to 
the invention, the calibrating disc comprises at least three marker sites that are arranged 
in a circular arc of the calibrating disc of essentially 0°, 120°, and 240°, in order to 
calibrate three cameras. 

[0032] Furth e r advantag e ou s d e velopment s of th e inv e ntion ar e the subject of th e 
r e maining subclaims. 

[0033] Advantageous developments of the invention shall be illustrated in an 
exemplary fashion by means of the following drawings. 

[0034] Figure 1 shows a schematic side view of an apparatus according to the 
invention for application and monitoring of an adhesive trail. 

[0035] Figure 2 shows a perspective view of the apparatus according to the 
invention of flgure Figiire 1 . 

[0036] Figure 3 shows the travel path of the apparatus according to the invention 
for application and monitoring of an adhesive trail. 

[0037] Figure 4 shows another travel path of the apparatus according to the 
invention with regard to the switching of the relevant camera. 

[0038] Figure 5 is a view of a single image composed from three image strips 
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from three cameras for seam application guidance and online monitoring of sealing agent 
application at one edge of a component. 

[0039] Figure 6 is another view of a single image composed from three image 
strips from three cameras, whereby two overlapping components are being glued 
together. 

[0040] Figure 7 shows a schematic view of a calibrating device according to the 
invention for calibrating the individual cameras of the apparatus according to the 
invention for automatic application and monitoring of a structure to be applied onto a 
substrate. 

[0041] Figure 8 shows a top view with regard to the basic principle of seam 

tracing. 

[0042] Figure 9 shows a top view with regard to the principle of 3D positional 
recognition. 

[0043] Figure 10 shows a top view with regard to profile analysis. 

[0044] Figure 1 1 is a schematic side view of the apparatus according to the 
invention with projection facility. 

[0045] Figure 12 is a schematic top view of a projection applied to be circular. 

[0046] In the following, the design of the apparatus according to the invention for 
recognizing a structure to be applied onto a substrate is illustrated according to 
fiffwes Figures 1 and 2. 

[0047] Reference number 10 indicates the schematically shown apparatus m 
Figure 1 for application and monitoring of an adhesive trail.- In the center of the 
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apparatus^ according to the invention^ is arranged an application facility 1 1 by means of 
which an adhesive trail 20 is applied onto a substrate and/30 or onto a sheet of metal 30 
31 proceeding from right to left in SgrFigure 1.- Three cameras 12, 13, 14 are arranged at 
equal distances from each other in a circle around the application facility 1 1 , e ach of 
whieh-. Each camera is directed at the application facility 1 1.- As is evident from figure 
Figure 1, the axial longitudinal axes of the three cameras 12, 13, 14 intersect the axial 
longitudinal axis of the application facility 11 just below the substrate 30 such that . 
Thus, the focus of the individual cameras is arranged right around the area of the 
application facility 1 1, in particular^ on a circular line. 

[0048] In the inspection of the adhesive, either the application facility 1 1 with the 
cameras or the substrate 30 is moved , wh e r e by the . The adhesive trail 20 is 
simultaneously applied to the substrate 30 by means of the application facility 1 1 , and 
wher e by while the cameras 12, 13, and 14 monitor the applied structure.^ For this-the 
purpose , it is f e asible to move of inspecting the adhesive, either the application facility 1 1 
with the cameras or the substrate 30 is moved in order to apply the adhesive frail- trail 20 
onto the substrate 30 such as to follow a desired progression.- In the method according to 
the invention, a first camera then-determines a reference contou r and/or , a reference line, 
or reference edge in ajeading direction, as shown by camera 14 towards the left in the 
cas e shown, in Figure 1. In order to regulate the progression of the structure to be 
applied^ according to the reference contour, wh e reby t he images recorded by the first 
camera 14 are used to guide the application facility 1 1 in the application of the adhesive 
trailr 20. The adhesive trail 20_is applied by the application facility 1 1 simultaneous to 
the determination of the reference contour , wh e r e by th e . The application facility 1 1 is 
moved to the corresponding track and/or the corresponding adhesive trail progression 
according to the correction values determined by the first cameraT 14. Synchronous to 
this process, the adhesive application track is monitored by a second camera in a_trailing 
direction.- B y this means, the cameras that are being moved along can control the 
adhesive trail 20, while i Hhe adhesive trail 20 is being applied, according to the reference 
contour ^independent of the travel path-and -. The cameras monitor the quality of the 
adhesive trail 20 online .- In fifc -Figiire 2, the adhesive trail 20 progresses from left to 
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right which is shown as a continuous line.- The desired progression of the adhesive trail 
20 that can be applied to the metal sheet 3J_and/or substrate 30 as reference contour (for 
example by laser or be embossed) is shown to the right of the application facility 1 1 by 
means of a dashed line. 

[0049] Figure 3 th e n s hows illustrates the progression of the adhesive trail 20 as 
indicated by arrows 21 and 22 , whereby the direction m4k>r field of view of the thre e 
individual cam e ras is shown in thr e e sites. The fi e ld of view 23, 24, and 25 of the three 
individual cameras eae h!2, 13, and 14 is shown in three sites. The field of view 23, 24 
and 25 of the three individual cameras 12, 13, and 14 are indicated by a rectangle drawn 
with a continuous line 23, a rectangle drawn with widely dashed lines 24, and a rectangle 
drawn with narrow dashed linesT 25. As is evident from fiftwe Figure 3, the direction of 
the individual fields of view 23, 24 and 25 of the cameras remains constant at all times 
wh e r e by and only the whole apparatus is moved. 

[0050] Figure 4 shows illustrates another progression of an adhesive trail 20- 
wher e by it is indicat e d in e ach case, that illustrates which field of view 23, 24 and 25 is 
activer-ir (e. g., marked at "active"). For example, Figure 4 depicts which camera having 
has the corresponding field of view shown as a rectangle is- 27 (e.g. marked as " active") 
while traveling along the adhesive trail 20 for quality control purposes - Moreover, at 
least a second camera is active in ajeading direction for seam application guidance 
and/or the progression of the sealing agent track , though this is J not shown h e r e for 
r e asons of clarity of pr e s e ntation.} ^ However, it is e vid e nt that the cameras indicated to 
be active are arranged in availing direction and are switched automatically according to 
the progression of the adhesive trai l 20. Therefore, at least one camera is active at any 
time for seam application guidance and fine adjustment of the application facility 
according to the given reference contour in the leading direction, which is opposite to the 
taUitt gtrailing direction. 

[0051] Figure 5 then show s illustrates three image strips whieh- that each represent 
a relevant section and/or strip of anjmage of the three individual cameras 12, 13, and 14 
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of figw eFigurc 1 .- According to the method of the invention, each camera 12, 13. and 14 
records just-a strip of the image 32, 33, and 34 in order to reduce the amount of data 
accordingly such that the recording rate can be increased - These individual image strips 
32, 33 and 34 of the three cameras 12, 13, and 14 are then joined into an image, whereby 
the image recording occurs at defined fixed time intervals and independent of the robot 
control of the application facility.- For example, the cameras 12. 13. and 14 only record a 
strip of the image , wh e reby instead 32, 33 and 34, Instead of an image height of 480 
pixels^ an image height of approxT imately 100 pixels (100 image lines) is used.- By means 
of this partial scanning techniquerirje.g. partial reading-out of the image recording chip), 
only small data streams are generated such that . Thus, the image recording rate can be 
increased several-fold accordingly . ,_ Synchronous image recording and parallel image 
capture allow the three image strips 32, 33 and 34 , one below the other, to be composed 
into a single image - As a result, the three imag e s, i. e . th e thr e e image strips? 32, 33, and 
34 are correctly arranged and assigned with regard to location and time relative to the 
travel path of the application facility 11 and without further ade^ delay and can be 
processed accordingly. This sp e cific . Thus, the image recording technique therefor e 
facilitates simultaneous and parallel recording of individual camera images, which 
achi e v e s an incr e as e in increases the image recording rate fer-both Iw-guidance and/or 
regulation of the application facility 11 and for online monitoring of the adhesive agent 
applied, whereby the images of all cameras 12, 13, and 14 are stored in a sequence of 
images. 

[0052] Once the images of the three cameras 12, 13, and 14 are stored in a 
sequence of images, a parameterization of this-ajeference track is carried out as the 
subsequent step of teaching-in the reference adhesive trail - The robot travel path, robot 
travel time, direction, width, and quality of the adhesive trail 20 are used in the 
parameterization. This The parameterization results in a type of vector chain for the 
adhesive frail which allows to attain th e trail 20. The vector chain attains high image 
recording iterates and comparably short partial sections ( e.g., between 1 and 3mm). 
Vectorization has another advantage in that the adhesive trail_J20, being in the form of a 
vector chain, can be stored in a camera-transcending global coordinate system. 
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[0053] As is evident from flgur eFigure 5 , which shows a illustrates seam 
application guidance at an edge of a component, the seam inspection is carried out online 
in the middle strip of figureFigure 5, whereby the segment of a circle is the area, in which 
the first camera provides for monitoring of the adhesive. 

[0054] Application facilities andA)r robots work with an internal interpolation 
clock time of 12 ms, for example.^ The regulation of seam application guidance according 
to the lower strip of figr Figure 5 cannot occur more rapidly than this interpolation clock 
time. At- Therefore, at a maximal robot travel speed of 700 mm/s, this m e ans only that 
the path traveled in 12 ms is 8.4 mm.- Accordingly, if a correction value is determined at 
time point x, the correction can be made only at the next interpolation clock time-Ji.e. 
8.4 mm after determination of the valued As a result, the reference edge must be 
captured at least 8.4 mm ahead of the nozzle.- This area is covered by the arrangement of 
the optical sensor system and correction values are made available in due time by the 
rapid analytical cycle (e.g., < 5 ms). Sinc e the cameras Because the cameras 12, 13, and 
j_4 are attached around the application facility such as to b e 1 1 in a fixed in-position, the 
progression of the adhesive trail 20 changes , wher e by . Thus, the seam correction can 
th e r e fore be carried out in the first strip according to the first camera, in the second strip 
according to the second camera or in the third strip according to the third camera. 
Consequently, as described above with regard to the online monitoring of adhesive 
application, another camera becomes active for seam correction when the reference seam 
migrates from the field of view of one camera into the field of view of another camera. 

[0055] The bottom strip of fig jjgurg 5 shows a bright cross on the line 
perpendicular to the edge of the component right at the edge of the component that is 
used as reference edge for seam application guidance .— Paralleling this, the seam 
inspection for monitoring of sealing agent application is carried out in an online fashion 
in the middle strip 33 of figr Figure 5. 



[0056] If the adhesive trail 20 progresses out of the field of view of a camera, the 
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adhesive trail 20Js transiently in the overlapping area of the ranges of angles of the two 
cameras. If the adhesive trail 20 then progresses from the segment of the circular line of 
the one camera via the overlapping area to the segment of the circular line of another 
camera, an automatic switch is made from the one camera to the other camera.^ This is 
shown, in particular, in figureFigure 4 by means of the active fields of view 27 of the 
individual cameras. 

[0057] The advantages mentioned above are attained by the individual cameras 
forming a circular caliper whose center is formed by the application facility 11, whereby 
the search for both the reference edge and the edges of the adhesive trail 20 proceeds on a 
circular line directly around the application facility 11. For this purpose, it is essential 
that the individual cameras 12, 13, and 14 are directed at the application facility U., 
whereby the axial longitudinal axes of the individual cameras 12, 13 and 14 intersect the 
longitudinal axis of the application facility 1 1 . 

[0058] The illumination module (not shown here) for the apparatus according to 
the invention is made up of light emitting diodes ( LEDs), in particular infrared LEDs, 
ultraviolet LEDs ( UV LEDs) or red- green-blue LEDs ( RGB LEDst)^ In order to attain as 
little movement or blur as possible and/e^ have a high contrast in image recording, the 
LEDs can be flashedr-be^ . For example, short, strong pulses of current on the order of 
1.0 to 0.01 ms can be applied to the diodes.- In this context, light-emitting diodes capable 
of emitting light of various colors are particularly advantageous such that because the 
sensor design can be switched to other types of adh e sive and/or adhesives or other colors 
of adhesives without reconfiguration. 

[0059] A teach-in run and/or a teach-in of a reference adhesive trail is illustrated 
in the following paragraphs below . 

[0060] The teach-in process of the reference adhesive trail can be started by the 
user by marking the position of the adhesive traiK 20. This is sufficient for fully 
automatic recognition of apposition and ^direction of the adhesive trail 20 in the 
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subsequent camera images^-sine e because the image recording rate is sufficiently high 
and the individual images are recorded very shortly after one anothe r, for e xampl e (e.g., 
every 1 mm to 3 mmv}^ From the starting point, the adhesive is scanned image by image ? 
wh e r e by th e . The adhesive trail position and the adhesive trail angle detected in the 
current image are used for the upcoming image as a priori knowledge. This The a priori 
knowledge facilitates fully automatic capture of the adhesive trail 20 without a human 
being having to determine andA>r assess the image and/or the position of the adhesive 
trailr 20. As a result, the search area can be limited and/or adjusted. 

[0061] Figure 6 shew sillustrates a guidance of a^seam application in the 
overlapping area of two components, in particular particularly at a place at which where 
the two components abut.- The second camera shows the strip of the second camera read- 
out according to the-apartial scanning method, in which the position of the overlap of the 
two metal sheets is determined as preference contour and/or preference edge to guide the 
seam application.- The strip of the third camera, in which the applied sealing agent track 
is monitored in parallel to seam application guidance, is shown in the bottom strip of 
fig rFigure 6 - For this purpose, the segment of a circle is shown in the bottom strip, in the 
middle of which progresses the adhesive trail 20 as indicated by a circle.- The image 
recording strip of the first camera J_4_is shown in the top strip of fig rFigure 6. 

[0062] Figure 7 shews illustrates a calibration facility 40 in the form of a circular 
calibrating disc in ord e r to assign . The calibration facility 40 assigns to the individual 
cameras their- 12, 13 and 14 a scaling factor, thekan angle assignment, and the center as 
w e ll as the and radius of the search circle - The calibrating disc consists of individual 
form elements and/or , shown as dots 41 A that are arranged on a circular line-and . The 
form elements are spaced at an angle distance of essentially 10— ° from one another. 
Moreover, marker sites 42 are arranged at equal distance from each other in order to 
calibrate the three camerasT 12, 13, and 14. A compensation calculation is used to 
calculat e from th e coordinat e s d etermined by utilizing the coordinants of the centers of the 
individual dots, on the one hand, the scaling factors of the individual cameras 12,13 and? 
on th e oth e r h 14 and, the center as w e ll as and radius of the search area - The marker 
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sites are located at angles of 0°, 120°, 240° in the global coordinate system allow 
allowing the angle assignment and the corresponding fields of view of the individual 
cameras 12, 13, and 14 to be determined.- The field of view e£for the individual cameras 
i sl2, 13, and 14 are indicated , in particular, by the three rectangles 43, 44, and 45 shown 
in fifflf eFigure 7 , wher e by th e . The form elements 41 can correspond to the circular line 
of the circular caliper for detection of the adhesive trail 20. 

[0063] Figure 8 shews illustrates three strips around the application facility 1 1 
each by dashed lines that represent the read-out area for the partial scan of the individual 
camerasr 12, 13, and 14. The strip 3 1 of the first camera determines the reference edge 
35 in ord e r to control and/or r e gulat e that controls and/or regulates the application facility 
1 1 according to the progression of the reference edge - Image strip 3 1 is ther e for e facing 
in ajeading direction and measures the position of the reference edge and/or fold 35 such 
that the application facility 1 1 applies the sealing agent onto the track corr e ct e d c orrectly 
according to the-reference contour 35.- After correction of the-ajobot track with regard to 
the-a_coachwork position, the joining seam is recognized by driving to the-afirst position 
and activating the-a_seam application guidance - After release of the process ( e.g., a seam 
is recognized), the robot track continuously receives correction values that are 
perpendicular to the application direction taught-in.- In this context, the-a^capture area can 
be ± lSmm-aftd^ whereby the-aj-egulation area is < ± 1 mm - The communication 
between the image processing system and the robot system and/or application facility 
proceeds, for example, by means of a standardized Ethernet interface using an XML 
protocol.. In the trailing direction, the two image strips 32 and 33 ar e shown that intersect 
in the area of the sealing agent track 3021. 

[0064] The online monitoring of an applied adhesive trail shall be illustrated 
briefly in the followingv paragraphs. The application facility 1 1 shown in figure Figure 1 
applies the adhesive frail— trail 20 onto the metal sheet 30, wh e r e by th e 31. The 
application facility 1 1 is moved jointly with the cameras 12, 13, and 14 over the metal 
sheet 3031 and regulated according to the reference contour.^ However, a kinematic 
inversion is also feasible, irer -for example, the metal sheet 30-31 being moved and the 
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application facility 1 1 with the cameras 12, 13. and 14 being arranged to be fixed in 
position.- The applied adhesive trail 20 is determined and analyzed synchronously and in 
parallel by one of the cameras 12, 13, 14 on the circular line of the circular caliper 
illustrat e d according to figur e fshown in Figure 5 , wh e r e by each ). Each camera 12, 13, 
and 14 records only a strip of the image 32, 33, and 34 and joins these into a single 
image.- The image recording rate is increased in accordance with the data reduction 
attained by because each camera recording l2, 13, and 14 records only a strip of the 
image , wh e reby th e 32, 33, and 34. The individual image strips 32. 33, 34 in the joint 
image facilitate the synchronous-and A parallel^ as well as simultaneous capture of the 
three camera images , and wh e r e by th e . The individual images of the three cameras 12, 
13, and 14 can be assigned directly as a function of location.- As a result, seam 
application guidance and online monitoring of the adhesive trail 20 in real-time is 
feasible ihat- and achieves high accuracy at high travel speeds d-ue-ie because of the high 
image recording rate both in the regulation according to the reference edge and in the 
inspection of the applied adhesive trail 20. 

[0065] Figvure 9 shews illustrates the basic principle of 3D positional recognition 
which that is carried out prior to applying a_sealing agent. Sme e Because the metal sheets 
31, for example raw coachwork of vehicles, are not always positioned in exactly the same 
position by the supply -supplier of the technology and the position of the joining seams is 
associated with tolerances, gross adjustment and/or gross positioning of the apparatus 
according to the invention is advantag e ous. an advantage. For this purpose, the camera 
image fields are switched to ajarge image and/or , a standard size^ or ajiill image, which 
is indicated for each particular case by the dashed line(s) 51 and/er 52.- The standard 
camera image field 51 shows the expanded field of view of a camera reading- out just the 
corresponding strip 31 according to the partial scanning procedure.- The strip 32 is scaled 
down analogous to this and according strip 52 and according to the ratio te-of the 
standard camera image field 52 - For example, the image strip 31 and/or 32 is scaled 
down by a suitable software-te, for example, to half ks- the width and 1/5 of te -the heightr 
of the image fields 51 and 52. For reasons of clarity of presentation, the corresponding 
standard camera image field 53 with corresponding image strip 33 is not shown-hefe^ In 
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the 3D positional recognition, an arbitrary feature 60 within the overlapping field of view 
of the two camera image fields 51 and 52 is measured.- Since the two camera image 
fields 51 and 52 overlap in the area of feature 60, the procedure of stereometry can be 
used in ord e r to facilitate a three-dimensional analysis, for example, of a hole or an edge 
of the component.- If, for example, a seam of two components has been recognized, the 
application facility 1 1 can carry out an automatic correction of position through the help 
of the sensory system in order to carry out the correction of the robot track and/or 
application facility track in an online fashion, as described in fig rFigure 8.- The sam e 
sensory system can carry out at the same time the quality control of seam sealing in an 
online fashion, as is al s o described in fi^ Figure 8. This- The 3D positional recognition 
provides for positional correction of the application feettit yfacilityl 1 and online 
regulation of the progression track of the application facility 11 as well as online 
monitoring of sealing agent application with a single sensory system that is mad e 
un configured , for example, e fwith three cam e ra s cameras 12, 13 and 14 in a fixed position 
arranged around the application facility such as to b e fix e d in position. ! 1. In this context, 
just-a strip of the image 31, 32, 33 is recorded according to the invention in order to 
utilize small data streams to achieve an increase in the image recording rate - The use of 
the partial scanning technique thus provides for an image refresh rate of approxr imately 
240 Hz or lower j ess. The images are therefore recorded at defined fixed time intervals 
and are independent of the speed of the robot and/ef the application facilitvr 1 1 . In 
addition, a frame grabber board (= -e.g., a personal computer ( PC) board for capturing 
images from the camera) is used in the analytical PC allowing images from all three 
cameras 12, 13, and 14 to be captured synchronously and in parallel.- The images are 
subsequently composed into one image ( i.e., 3 strips one below the other) providing the 
advantage of three images each being directly assigned accordingly in a location- 
dependent fashion. 

[0066] Moreover, it is sufficient, in particular, to search for and analyze, for 
example, the sealing agent track in one of the three images. Jf the angle value exceeds a 
certain value, an automatic switch to the neighboring camera is made - In this context, the 
angle value refers to a full circle of 360°, which results in a global coordinate system. 
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Each camera comprises an area of overlap with the next camera - The selection of the 
camera is made independent of the position of the application facility and/or l 1 a robot 
position^ and /or indep e ndent of a time component, but rather always refers to the actual 
inspection results which are captured in the global coordinate system.- This prevents 
errors that are generated by the relative inaccurate robot controls and/or application 
feeUity facilitv 11 controls. 

[0067] The three-dimensional profile analysis by means of a projection is 
described according to ftgwesFigures 10 and 11 , in order to provide for a positional 
correction of th e application facility as d e scrib e d abov e according to figur e 9. For r e asons 
of clarity of pr e s e ntation, fig. . Figure 10 again shows only illustrates by dashed lines two 
camera fields of view 51, 52 indicat e d by dash e d lines. ^ In the overlapping area of the 
two camera fields of view 51, 52 are shown a plurality of laser lines 60 that are used for a 
profile analysis with regard to the width and contour of structure lines and for generation 
of so call e d so ft contours - The laser lines 60 are generated by a projection facility that 
can, for example, be arranged on the optical sensor with three cameras - Moreover, the 
projection facility can ju s t as w e ll be arranged directly on the application facility 11.- The 
sensor with the three cameras is shown schematically by the circle 70 - The laser lines 
and/60 or laser strips are projected onto the component 30 and/ 32 or metal sheet 
highlight 3 1 highlighting contours on the component 30 that cannot be used for ajhree- 
dimensional analysis by conventional image processing.- Artificial features are generated 
by means of the laser lines 60 on the component 32 and can subsequently be analyzed by 
means of image processing according to stereometry - Thus, figweFigure 10 shows the 
principle of three-dimensional positional recognition prior to the application of the 
sealing agent in case no hard, analyzable features are present.- In contrast, a hard contour 
is described by means of feature 60 in fig. 9 as m e ntion e d as illustrated by Figure 9 and as 
described above. 

[0068] Figrure 1 1 shews illustrates a side view of the application facility 1 1 with a 
sensor unit 70 attach e d to it, whereby th e 71. The sensor TQ unit 71 , aside from the three 
cameras, can comprise at least two projection facilities 61 that project laser lines onto the 
metal sheet and/31 or the substrate 30, as is-shown schematically by the dashed liner 62. 
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Arranging multiple projection facilities 61 around the application facility 1 1 allows a 
gap-free contour to be generated on the metal sheet 30, wh e r e by th e 31. The gap-free 
contour can be used for three-dimensional analysis sine ebecause the sensor 71 and the 
projection facility 61 are calibrated - Accordingly, fife -Figure 1 1 shows two projection 
facilities 61 in an exemplary fashion - Proje ction facilities 61 of this type can, for 
example, project a laser onto the substrate and/or the component or can b e made up of . 
In addition, projection facilities 61 can have an LED module comprising an adapter lens 
to generate a line on the substrate. 

[0069] The projection facilities 61 can be used both for three-dimensional 
positional correction prior to sealing agent application as well as for online analysis of a 
height and ajrofile of the applied sealing agent.- For three-dimensional positional 
correction, the projection facilities 61 can, preferably^ project multiple lines.- For height 
analysis, one or more projection facilities 61 should be provided that project a line or, as 
shown in figr Figure 12, a circular contour onto the component and/or substrate. In this 
context, multipl e Multiple lines can be advantageous for the analysis. 

[0070] As shown in fig. 12, it is also f e asibl e Figure 12 illustrates to determine 
sealing agent height and/or sealing agent contour and sealing agent position according to 
the principle of triangulation by means of the image processing simultaneously an4 
directly after sealing agent application. For this purpos e , a A round contour 63, for 
example, is applied to the metal sheet 3031 by the projection facilities , wh e r e by the . The 
sealing agent and/20 or the sealing agent track 2021 provides for a change of height and 
position of the projected contour 63- The round projection contour 63 thus changed m 
turn-is determined by the individual object fields of the individual camerasr^^e- 12, 13 
and 14. Thus, the original shape of the projected contour 63 is thus-deformed by the 
sealing agent 20 such that the width, contour, but also the height and position of the 
applied sealing agent 20 can be determined according to the principle of triangulation.- In 
the principle of triangulation, there is a defined angle between the camera and the 
projection facility 61 , whereby the camera and the projection facility facility 61 are 
calibrated with respect to each other- Because of the angle, the substrate contours 
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illuminated by the projection facility appear in different positions on the light-sensitive 
chip and/or charge-coupled device ( CCD) chip or complementary metal oxide 
semiconductor ( CMOS) chip as a function of their height such that the height and contour 
of the sealing agent can be calculated due to the calibration of the camera and projection 
facility 61. 

[0071] According to an embodiment that is not shown here, the sensor 71 that is 
mad e up, in particular, o f configured with three cameras 12, 13, and is!4 arranged around 
the application facility is mad e up l 1, which is configured such that the optical axes of the 
individual cameras 12, 13, and 14 are directed such as to be parallel to each other ? 
wh e r e by th e . The cameras 12, 13, and 14 are, in particular, dir e ct e d such as to b e 
perpendicular to the substrate aftd/30 or metal sheet v 31. An arrangement of this type 
allows the sensor 71 to be arranged particularly close to the area of sealing agent 
application, whereby the fields of view of the individual cameras 12, 13, and 14 comprise 
an overlapping area whose size depends on their wide angle. 
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METHOD AND APPARATUS 
FOR AUTOMATIC APPLICATION AND 
MONITORING OF A STRUCTURE TO BE 
APPLIED ONTO A SUBSTRATE 

ABSTRACT OF THE DISCLOSURE 

A method and apparatus are provided for automatic application and monitoring of 
a structure to be applied onto substrate. A plurality of cameras positioned around an 
application facility are utilized to monitor the automatic application of a structure on a 
substrate by means of a stereometry procedure. Three-dimensional recognition of a 
reference contour position results in the overlapping area to be used for gross adjustment 
of the application facility prior to applying the structure. 
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